We propose a new regularization method based on virtual adversarial loss: anew measure of local smoothness of the output distribution. Virtual adversarialloss is defined as the robustness of the model's posterior distribution againstlocal perturbation around each input data point. Our method is similar toadversarial training, but differs from adversarial training in that itdetermines the adversarial direction based only on the output distribution andthat it is applicable to a semi-supervised setting. Because the directions inwhich we smooth the model are virtually adversarial, we call our method virtualadversarial training (VAT). The computational cost of VAT is relatively low.For neural networks, the approximated gradient of virtual adversarial loss canbe computed with no more than two pairs of forward and backpropagations. In ourexperiments, we applied VAT to supervised and semi-supervised learning onmultiple benchmark datasets. With additional improvement based on entropyminimization principle, our VAT achieves the state-of-the-art performance onSVHN and CIFAR-10 for semi-supervised learning tasks.
展开▼
